Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More realistic benchmarks #639

Merged
merged 7 commits into from
Oct 11, 2021
Merged

More realistic benchmarks #639

merged 7 commits into from
Oct 11, 2021

Conversation

ethanfrey
Copy link
Member

@ethanfrey ethanfrey commented Oct 8, 2021 •
edited
Loading

More work on #635

  • Unify setup (exact same chain config for bank and cw20 sends)
  • Parameterize DB (memdb / leveldb)
  • Parameterize Initial Accounts (how many inactive accounts fill up space before benchmarking)
  • Send tokens to different accounts each time (still one sender)

@ethanfrey ethanfrey requested a review from alpe as a code owner October 8, 2021 20:27
@ethanfrey ethanfrey marked this pull request as draft October 8, 2021 20:27
@ethanfrey ethanfrey mentioned this pull request Oct 8, 2021
Closed
6 tasks
@ethanfrey ethanfrey marked this pull request as ready for review October 8, 2021 21:21
@ethanfrey
Copy link
Member Author

Please check out results from the last commit where I filled out equal state for bank and cw20.

goos: linux
goarch: amd64
pkg: github.com/CosmWasm/wasmd/benchmarks
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
BenchmarkTxSending/basic_send_-_leveldb-36         	    1029	   1221697 ns/op
BenchmarkTxSending/cw20_transfer_-_leveldb-36      	     840	   1377302 ns/op
BenchmarkTxSending/basic_send_-_leveldb_-_8k_accounts-36         	     760	   1586794 ns/op
BenchmarkTxSending/cw20_transfer_-_leveldb_-_8k_accounts-36      	     819	   1573988 ns/op
BenchmarkTxSending/basic_send_-_memdb-36                         	    1152	   1221075 ns/op
BenchmarkTxSending/cw20_transfer_-_memdb-36                      	     864	   1349733 ns/op
PASS
ok  	github.com/CosmWasm/wasmd/benchmarks	27.370s

It seems like the cw20 is 10~20% slower in the most unrealistic case (memdb, no other accounts).
And statistically equal in the most realistic of these (leveldb, 8k accounts).

@ethanfrey ethanfrey changed the title Pull out test initialization More realistic benchmarks Oct 8, 2021
@ethanfrey ethanfrey merged commit 6a471a4 into master Oct 11, 2021
@ethanfrey ethanfrey deleted the 635-better-app-benchmarks branch October 11, 2021 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alpe alpe Awaiting requested review from alpe

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ethanfrey